
SYSTEM AND METHOD FOR INTERPOLATING A TARGET IMAGE 

FROM A SOURCE IMAGE 
The present invention relates to the field of image displays, and more particularly to a 
system and method for improved interpolation in digitized image displays. 

BACKGROUND OF THE INVENTION 

Traditional raster display devices such as television monitors, video displays have offered 
few display option features such as image zooming, picture-in-picture and such like. 
However with the increase in digitization of video images (here is an expectation that 
raster display devices offer such features. In the case of image zooming, the resolution of 
the image being displayed is seldom limited by the raster display resolution capabilities, 
but by the source image pixel resolution. Accordingly, many techniques have been 
implemented to improve the perceived resolution of digitized images, both to support the 
additional features demanded by users and to improve the overall clarity of the perceived 
image to the user 

A digitized image consists of a rectangular array of pixels having Rs row pixels and Cs 
column pixels. During image enlargement the information in a source image is used to 
create a new target image having RT row pixels and CT column pixels where typically 
Rs<=* RT and Cs<= CT. That is, more pixels per row and per column are needed to 
satisfy the requirement of the new target image. Target pixels are usually generated by 
using interpolation, look-up tables or other known techniques. 

Referring to Figure 1 there is shown a two-dimensional array of source pixels (circles) 
25 and an anticipated target pixel (x) to be generated in a target row between successive 
rows of the source pixels. One way to generate the target pixel x would be to take one 
quarter of the value of the immediate neighbors and sum the result. However, it is 
difficult to decide on the ideal combination of source pixels and their associated weights. 
"Ideal" in this case depends on the strategy, rule, algorithm or method implemented to 
30 ensure that the enlarged image is most visually pleasing. 
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Often when computing the target pixeU in the absence of more suggestive information 
about the nature of the pixel data in a row, an assumption is made that the source pixels 
that are physically close to one another contain information that is the most similar. And, 
by extension, it is often assumed that the closer the source pixel is to the target, the more 
5 meaning it can impart on the final value of the target pixel's intensity. For example US 
Patent No, 5,703,968 describes a method and apparatus for detecting and computing the 
level difference between pairs of pixels surrounding the target pixel on an upper and 
lower line. The level differences are compared with each other and a line having a 
smallest level difference is detected an effective interpolation line information is 
10 generated. Similarly, in U.S. Patent No. 5,991,463 there is described a multi point 
'filter" interpolator, or four-tap filter wherein interpolated unsarnpled target pixels are 
calculated as a function of four source pixels and a parameter representing the distance of 
the desired pixel from a source pixel. 

,q 15 This technique is limited however when applied to a solid diagonal line that has been 

Hi digitized and displayed on a raster-imaging device. Closer inspection of the components 

fy that comprise the diagonal line reveals that the line is made up of a series of shifted 

jjp. horizontal line segments as shown in Figure 2(a). In many applications, and in image 

s is? 

3 enlargement in particular, it is very important to identify the horizontal (vertical) offset 

O 

fk 20 between rows (columns) so the data in the image can be properly interpreted. 



Although the problem of maintaining the perceived resolution of an image using a 
corTelation-based algorithm is applicable to progressive scan and interlaced images, its 
proper use most strongly influences the quality of an image when an image is interlaced 

25 such as is the case in interlaced video. For instance, an interlaced image frame of 2N 
lines has two fields of N lines per field. When the image content is static in both fields, 
the effective resolution is 2N lines. But when the image content is in motion, the effective 
resolution drops to N lines in general, due to the temporal latency between fields. In 
regions where the image content contains less detail, the loss in resolution is less 

30 perceptually significant In regions where the image content has detail, i.e. where the 
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correlation function between row or column pixel data is sensitive to a spatial 
displacement, the reduction in perceived resolution can be quite noticeable. 

For example as shown schematically in figure 2(a), if we were to interpolate vertically by 
taking a linear combination of the pixel immediately above and immediately below the 
gaps between the lines in Figure 2(a) t we would end up with a deintcrlaced (or scaled) 
image as shown in Figure 2(b). Clearly this is an unacceptable image quality. Thus, the 
different ways in which data is interpolated has a large impact on the perceived resolution 
of the image. 

Accordingly, there is a need for a system and method that allows the deinterlacing of 
images for enlargement while maintaining its perceived resolution when redisplayed. 



SUMMARY OF THE INVENTION 

*Q 15 The present invention seeks to provide a method of interpolation for maintaining the 

PJ 

^ perceived resolution of a digitized image when the image is enlarged. 
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ry In accordance with the invention there is provided an interpolator for processing an 

g image having an array of pixels, the interpolator comprising a feature extractor for 

83 20 processing a pixel sequence contained in the array of pixels to extract visually significant 
Ijj features therein; a correlator for determining similarities between the extracted features in 

0 adjacent pixel sequences and; an alignment controller for generating a position of a target 

pixel based on the output of the correlator, such that the interpolated target pixel is 

generated from the visually most relevant source pixel data. 

25 

In a preferred embodiment the feature extractor is implemented with a state machine. The 
features are identified by observing pixel sequences of row or column pixel data. The 
method of interpolation is similar to identifying regions of correlation between pixels, but 
it does so by first extracting features and then determining whether those features belong 
30 together. The similarity, or correlation, between extracted features is due to a visually 
significant structure, such as a for example, a line, edge or ramp, of arbitrary orientation. 
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The interpolation process uses feature-based comparison to maximum visual benefit and 
includes a process for identifying and storing specific characteristics of an image for later 
use. 



5 BRIEF DESCRIPTION OF THE DRAWINGS 

An embodiment of the present invention will now be described by way of example only 
with reference to the following drawings in which: 
Figure 1 is a schematic diagram of a pixel array; 
Figures 2(a) and (b) are schematic diagrams of a digitized diagonal line; 
10 Figure 2(c) is an interpolated image of the line using a vertical filter interpolator 
according to the prior art; 

Figure 3 is a flowchart of the process of interpolation according to the present invention; 
Figure 4 is a schematic diagram of the digitized diagonal line of Figure 2(a) generated in 
accordance with an interpolation process of the present invention; 
43 15 Figure 5 is a schematic diagram showing increasing intensity values in a pixel sequence 

characterizing an upward ramp feature; 
fU Figures 6(a) and (b) are schematic diagrams of a flow chart and bubble diagram for a 

state machine for processing an upward ramp feature; 

3 Figure 7 is a schematic diagram showing intensity values in a pixel sequence 

O 

p 20 characterizing a level segment feature; 

Figures 8(a) and (b) are schematic diagrams of a flow chart and bubble diagram for a 
state machine for processing a level segment feature; 

Figures 9(a) and (b) show a flow chart and bubble diagram for implementing downward 
Figures 10(a) and (b) are graphical representations of an intensity profile in adjacent 
25 lines of pixels in an image; 

Figure 11 is a flow chart showing a matching algorithm; 

Figure 12 is a schematic diagram showing the interpolating process using a pivot pixel; 
Figure 13 is a schematic diagram showing sub-pixel interpolation using a target pixel; 
Figure 14 is a flow chart showing a process for computing relative shifts; and 
30 Figure 15 is a schematic diagram of an interpolator according to an embodiment of (he 
present invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following description like numerals refer to like structures in the drawings. The 

schematic diagram of a pixel array shown in Figure 1 will be used in the following 

5 description to more clearly illustrate the concepts of the subject invention Referring now 
to figure 3 there is shown a generally at 300 a block diagram of an interpolator according 
to an embodiment of the present invention. The interpolator 300 comprises a feature 
extractor 302 for identifying visually significant features in a pixel sequence contained in 
an array of pixels 304 and based on predetermined threshold criteria 308, a feature 

10 comparator 306 for generating a correlation between the extracted features in adjacent 
rows or columns of pixels and an alignment controller 310 for determining pixels to be 
used in generating the target pixel T based on the output of the feature comparator 306, 
such that correlation is used to maximum visual benefit in computing the value of the 
^ target pixel In the following description each component of the interpolator 300 is 

15 described in detail along with worked examples of their operation. 

rtj 

n j Referring back to figure 2(a) there is shown a schematic diagram of a portion of an image 

03 200 containing a solid diagonal line that has been digitized and displayed on a raster- 

fU 

s imaging device shown schematically in Figure 2(b). The image generally consists of an N 

| 20 by M pixel array. Closer inspection of the component pixels that comprise the diagonal 
M line reveals that the line is made up of a series of shifted horizontal line segments 202, 

' 203, 204 and 206. In many applications, and in image enlargement in particular, gaps 

between the lines have to be filled-in. It is important to identify the horizontal (or 
vertical) offset between rows (or columns) so the data in the image can be properly 
25 interpreted by the viewer. During interpolation, for example, the perceived resolution 
will be better if the orientation of the line is known and an interpolation filter is aligned to 
take advantage of the image or feature orientation. 

The present invention provides an improved interpolation system that uses pixels whose 
30 information content is the most alike, and not necessarily pixels that are physically close. 
Accordingly, if a filter is applied to the original pixel with an understanding that pixels in 
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subsequent rows are misaligned by five pixels horizontally, computing the intermediate 
row by interpolation results in the image shown in Figure 4. Clearly this directional 
interpolation method results in an image that recovers the digitized line more faithfully 
than the interpolation used to derive the image shown in Figure 2(c). The perceived 
resolution has been maintained during deinterlacing (enlargement). This requires that the 
relative shift in the feature of interest in the image is determined. That is, pixels whose 
content is alike* or more precisely, belong to the same feature, must be identified and 
connected during interpolation. In order for this to happen, many difficult problems must 
be overcome and as will be described below. 



For the purposes of the present discussion we will assume that as shown in Figure 1, 
pixel data enters from the top right, P(0,3), and exits at the top left P(0,0). Once a row of 
pixel data has passed across the top row, it is re-circulated and appears on the second 
0 row, P(l,*), again entering from the far right, P(l,3), and exiting at the left, P(1,0). In 

■II 15 this way, a row of pixel data, held in a line store register, becomes the next row of pixel 
H data after a one-line delay. 

rU 

jjj In order to generate the feature information, a difference circuit component 301 computes 

^ the change in intensity between adjacent pixels in the same row. That is, A=P(Ri,Cj) - 

O 

03 20 P(Ri,Cj.i), where R* is the ,-th row and Cj is jtb column. Thus, for the first two pixels in 
jt rowl: A=P(0,1>P(0,0). 

Q 

The value of A is used throughout the feature extraction process. 



25 The feature extractor 302 (FE), or as it is also known. Feature Identification, performs a 
process whereby specific characteristics of the image are identified and which may be 
recorded for later use. When enlarging an arbitrary image, the most important features in 
an image, on a line-by-line basis, are usually, but not limited to, a ramp (a succession of 
either increasing or decreasing intensities), edges (a large change in intensity, sometimes 

30 called a step) and a level segment (a series of successive intensities mat are relatively 
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constant). There are other features such as noise, spikes and so on, which can also be 
identified and stored for subsequent use. 

In one embodiment a state machine (SM) is used to detect a specific feature. The targeted 
5 feature for extraction is user-definable and, therefore, programmable, so alternative 
definitions of specific features can be changed in a dynamic manner. In the following 
description, wc will restrict the discussion of feature extraction to a row of data, but it is 
acknowledged that the method described herein applies equally well to column data. 

10 In Figure 2(b) a single row of pixel data may be used to illustrate the operation of a state 
machine. The row of pixel intensity data 202 in Figure 2(a) has three components: a 
downward segment (white to black ramp), a level segment (Black-Black) and an upward 
segment (black to white). Here the terms "downward" and i4 upward M are merely used to 
O describe increases or decreases in pixel intensity. A state machine is used to extract 

15 specific portions of the intensity data. Separate state machines are used to identify 

W specific features such as level segments and upward ramps. Other segments may also be 

yy 

y identified, however, for illustrative purposes, the following discussion will be limited to 

the segment types described above. 
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*>q 20 The state machine uses basic hardware components such as adders and comparators to 
perform the feature extraction operations. The state machine flow control decisions use 
p these components in any way, thereby rendering the state machine fully programmable. 

Thus, alternative flow control algorithms can be programmed in the state machine to look 
for level segments, or other features of interest, in a flexible manner. 

25 

In general, if N different features are to be identified, N state machines are required, 
although depending on the precise definition of the features will need to be identified, 
fewer state machine may be needed. The state machines are independent and operate 
concurrently so the approach lends itself to easy expandability. Adding more state 
30 machines, as required, it easy within the current framework provided it is accompanied 
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by the necessary hardware. This approach also lends itself easily to a software 
implementation. 



Referring to Figure 5 there is shown a sequent of pixels 510 and a trajectory of its 
5 intensities 506 characterizing an upward ramp. The trajectory 506 is bounded above and 
below by thresholds 502 and 504. These thresholds are user-defined. For the purposes of 
discussion, we can define an upward ratnp so that it must satisfy the constraints: 



a. successive intensities must be increasing by a positive minimum 
10 threshold; 

b. the above must hold for some minimum number of pixels; 

c. there may be a finite number of exceptions to (a) 

d. the trajectory of intensities must be contained within an upper and a lower 
□ threshold; and 

yij 1 5 e. there may be a finite number of exceptions to (d). 

W 

Referring to Figure 6(a), there is shown a flow chart of state machine for implementing 
an upward ramp 506. 



^0 



03 



O 



20 Nu is a user-defined parameter that sets the maximum number of violations permissible 



N before the candidate upward segment is rejected. A violation, in the context of the 

IH 

flowchart, is a set of pixels that do not meet the criterion: p(i)-p(i-l) > Tup; Tup is a 



user-defined parameter that defines the threshold value for a upward step; 



25 Referring to Figure 6(b), there is shown a bubble diagram for implementing the flow 
chart of Figure 6(a). The states have the following behaviour for the upward ramp state 
machine. 

State 0 

N=0 
30 Ps=p(i) 
is=i 
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State 1 

i=i+t 

State 2 

5 Save the values that con-espond to the starting point (is,ps) and ending point (ie.pe) where 
ie=j-l and pe=p(i-l) before returning to state 0. 
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Note that Nu is the number of consecutive upward steps needed to qualify as a ramp. 
Tup is the size of each step. 



In a similar way in which the ramp was defined above, we can define a level segment. 
Although there is no unique way to define a Level segment, an example of one definition 
CS is given below. 



f J 1 5 Referring to Figure 7(a) there is shown a typical row of pixels defining level sequent 702, 
and accompanied by a threshold plot 704. In general, a level segment may be determined 
by applying the following list of criteria: 



Q (a) the locus of intensities (the intensity trajectory) following a staring point (ps) 

pi 

y 20 must lie within a band defined by (ps-TV, ps+TV) for at least NL pixels, where 

^ ps is the potential start location of the level segment; TV is a user-defined 

0 

y, violation threshold (typically this is set to about three and is useful to counter the 

effects of noise); 

(b) there may be at most NT violations of the band threshold. Here NT is a user- 
25 defined threshold that places a maximum number of threshold violations of 

condition (a); 

(c) there may be no more than NCT consecutive intensity values beyond the 
threshold band defined in (a). NCT is a user-defined threshold that places a 
maximum number of consecutive allowable violations before the candidate level 

30 segment is rejected; and 
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(d) if at any time the trajectory of intensities ventures beyond the confines of the 
threshold band defined by (ps-TD, ps+TD) the candidate level-segment is ended. 
TD is a user-defined threshold defining the permissible region in which a level 
segment must lie as defined by ps before it is rejected (disqualified). 

5 

The band defined in (a) above provides the ability to build in a flexible forgiveness 
factor. This is useful in the event the intensity values are corrupted by noise and more 
noise immunity is required. If the locus of intensities has satisfied all constraints, then a 
Level feature is deemed to have occurred. Its starting (is) and ending (ie) locations and 
1 0 starting and end intensities (ps) and (pe) are stored for later analysis. 

Referring to Figures 8(a) and (b), there is shown a flow diagram 800 and a bubble 

diagram 810 for implementing a Level segment feature extraction according to an 

Q embodiment of the present invention. In Figure 8(b), the following operations take place 

yg 15 in each of the states: 
ffl 



H J State 0 

H cv=o 

f% 20 is*=i 

03' 

|4 State 1, 2, and 3 

u 

25 State 4 

0=Cv+l 



State 5 

Store the starting point location and ending locations is and ie, and the respective starting 
30 and ending pixel intensities ps and pe. 
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Referring to Figures 9(a) and 9(b), there is shown a flow chart and bubble diagram for an 
algorithm executed by a state machine used to detect the presence of a downward 
segment. The following parameters are used in the diagrams. 

5 

• NV is a user-defined parameter that sets the maximum number of violations 
permissible before the candidate downward segment is rejected. A violation, in 
the context of the flowchart, is a set of pixels that do not meet the criterion: p(i)- 
p(i-l)>Tdown; 

10 • Cv is the count that contains the current number of violations. 

• Tdown is a user-defined parameter mat defines the threshold value for a 
downwards step; 

^ • Ndown is a variable that contains the number of downwards steps taken in the 

J current candidate downward segment. 

S 15 

W 

^ The upward trend state machine uses the same logic, except the polarity of the thresholds 



ru 



25 



and comparisons is reversed. 



$ The states in figures 9(a) and 9(b), have the following behaviour for the downward ramp 

CP 

20 state machine. 

q State 0 

H> N=0 



Ps=p(i) 
is=i 

State 1 

H+l 



State 2 

30 Save the values that correspond to the starting point (is,ps) and ending point (ie,pe) where 
ie=i-l and pe=p(i-l) before returning to state 0. 
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Note that Nd is the number of consecutive downward steps needed to qualify as a ramp. 
The magnitude of Tdown is the size of each step. 



The operation of the interpolator 300, can be more clearly understood by referring to a 
5 specific example. Referring to Figures 10(a) and 10(b), there is shown a series of pixel 
intensities that correspond to image segments in Figure 2(b). 

The Feature Extractors (FE) 302 processes pixel data arranged in a two-dimensional array 
or matrix having elements P(i j). Each row in the matrix is denoted by P(i,*), - a one- 
10 dimensional sequence of intensities similar to those shown in Figure 2(b). 



The Feature Extractors 302 log the Downward, Level and Upward segments to a feature 
table (Table 1) for rows 1 and 2, where the number represent intensity values on an 

•fj arbitrary scale of 0 to 255. 

• fk 

r, 15 Table 1 





SI 


El 


S2 


E2 


S3 


E3 


S4 


E4 


Intensity 
(Row 1) 


255 


5 


5 


5 


5 


255 






Intensity 
(Row 2) 


255 


255 


255 


5 


5 


5 


5 


255 


Position 
(Row1) 


6 


19 


19 


25 


25 


42 






Position 
(Row 2) 


3 


17 


17 


30 


30 


36 


36 


53 



The elements in the table comprise a pair-wise grouping of numbers (start position SI, 
start intensity El) and (end position S2 S end intensity E2) in Table 1 correspond to a 
feature that has been extracted and logged to the Feature Table. For example in row 1, 
20 positions 7 through 20 correspond to a downward ramp. Each time a feature is identified 
in the source data, it is logged to the Feature Table. Should the Feature Table become 
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full, a "Feature Table Full" Hag will be 5 eL Usually, eight (8) bits are needed to represent 
intensity data and eleven (11) bits are needed for the pixel positioning. These numbers 
are format dependent in general. 




5 Once the feature table is compiled, the Feature Comparator 306 attempts to match like 
features held in two adjacent rows in the Feature Table 1 . 

After the first row of pixel data has passed, all features of interest have been extracted 
and logged to the Feature Table. Immediate thereafter, the second row of pixel data 
10 arriving at P(0,2) is examined and the features it contains are extracted. And at the same 
time, the Feature Comparator 306 is attempting to match like features. If a match is 
found, it is stored in a Matched Table (Table 2). The information in the Matched Table is 
used later on by the Alignment Controller 3 1 0. 

15 The operation of the Feature Comparator 306 may be understood by comparing the set of 
intensities Figure 10(a) with those of Figure 10(b). Figure 10(b) shows the intensity 

HI profile on row 2 which is one line store in advance (earlier in time) of row 1. Table 2 

W 

jy shown the corresponding extracted feature information. 

fa 

gjj 20 The Feature Comparator 306 implements an algorithm that attempts to determine whether 
^ rows '0 1 and T are correlated, and further, which segments or features belong together 

Q (constitute a match). Clearly the pixel data in row % 0' and row 4 V is correlated, since their 

intensity profiles are very similar except for the horizontal positional shift. Some 
restrictions may be placed on the search so that only segments within a window of N 
25 pixels are compared. 



y3 



Referring to Figure 11, there is shown a flow chart of a matching algorithm 1160 
according to one embodiment of the present invention. The matching algorithm may be 
described as follows: Let P(0,i), P(lj), l(0,i) and 1(1 j) represent the pixel position and 
30 pixel intensities for rows 0 and 1, respectively. Let the window N size that limits the 
search region be equal to twenty five (25) pixel. Then the possible matches for the 
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segment (S1,E1) from row 1 are (S1,E1), (S2,E2) and (S3.E3) from row 2 as Sl(row 1)_ 
S2(row 2) < 25. To determine whether a match exists, each pair of intensities must match 
to within a chosen tolerance T. If T = 20, then clearly, ab$[I(0,0)-I(l ,0)] and abs[I(0,i)- 
KU)] > T so no match exits for these segments. The next candidate segments for reveals 
that abs[I(0,l)-I(U )] and abs[I(0,2)-I(l,2)] < T so there is a match. 



10 



To ensure that the nearest matched pair has been found, another search must take place 
over the alternate row keeping the segment in row 2 constant and finding the nearest 
matching segments in row 1 . If another match is found, then the nearest positional match 
is deemed the match. 



%4 
fJJ 

ru 

61 



o 



15 



20 



25 



It is not difficult to extend the matching algorithm to include three rows (columns) of 
pixel data. In addition, a predictive circuit can be employed that estimates the next 
correlated feature based on the previous two matches. 

The matching indices are stored in the Matched Table as shown in Table 2. Table 2 
contains paired indices of matching segments for Table 1. 
Table 2 





Position Row 1 


6 


19 


19 


25 


25 


44 




Position Row 2 


17 


30 


30 


36 


36 


53 



30 



The matching algorithm finds the initial bearing of the segments in the Feature Table. It 
must be run at the onset of new row data or when the trend bearing is lost. Once the 
bearing has been established, it is possible to match segments without resorting to a two- 
sided iterative search. As long as trend segments are properly tracked, the bearing 
portion of the match need not be invoked. Matched segments are removed from 
consideration in subsequent matching. 

In general, Table 2 will contain one extra bit of information indicating whether or not a 
region corresponds to a non-transition segment and possibly information needed for sub- 
pixel interpolation. Sub-pixel interpolation is explained later. In our example, such an 
overlap is absent 
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Once the features in adjacent rows are matched, an alignment controller (AC) 310 
computes the sequence of relative horizontal shifts that are needed between adjacent rows 
in order to bring matched transition segments into alignment. The aligned segments may 
5 then be processed using one of many standard interpolation methods or filters to 
determine the value of the target pixel. 



After the trend bearing is found, and the Matched Table is populated, phase information 
is used to compute the relative shift needed to align matching transition segments. In 
10 order to understand how the alignment controller computes the sequence of relative shits, 
we will need to introduce two terms namely: Transition Segment and Pivot Pixel. 

A Transition Segment (TS) is a segment that exhibits changes in intensity that is not a 
Level. Thus upward or downward ramps or variations thereof may be characterized as 
15 Transition Segments. A matched Transition Segment is distinct from a non-Transition 
Segment jti that it is only when such segments are actively participating in interpolation 
that the desired relative shift between rows is not necessarily zero. Alternatively, the 
desired alignment of a filter input is not necessarily vertical. 

|H 20 The pivot pixel (PP) is a pixel in the matched segment that defines the beginning or end 
H of a matched transition segment. For example, referring back to Table 2, the matching 

^ segments are A2 and B3 . A2 is the pivot pixel position because I0(i)=l < II (j)=l 1 . 



m 
m 

m 
ru 



Referring to Figure 12, there is shown a schematic diagram or a pair of adjacent lines of 
25 source pixel 1 202 and 1204. A line of target pixels 1206 is shown bounded on the top by 
the top pixel row 1202 and at the bottom by the bottom pixel row 1204. As may be seen, 
in order to generate a value for successive target pixel, the pivot pixel is used repeatedly 
with the bottom row pixels until the pivot pixel in the bottom row has been shifted into 
alignment with the top row pivot pixel. Following alignment the orientation of 
30 interpolator filter is maintained throughout the transition segment until either the next 
pivot pixel or a matched non-transition segment is encountered. 
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The operation of the alignment may be described as follows. A straight line is cast from 
5 the pivot pixel (PP(i)) through a target pixel (x). The line intercepts the adjacent source 
row at a bounding pixel. The bounding pixel location will not always coincide with a 
source pixel In order to generate the desired bounding pixel, a technique known as sub- 
pixel interpolation is used. 

Sub-pixel interpolation is used to generate an effective bounding source pixel where there 
is none. To generate such a pixel, interpolation is performed in a non-separable manner. 
This means that horizontal and vertical interpolation takes place concurrently. To better 
understand why sub-pixel interpolation is required consider the equation of a line 
originating at the pivot pixel PP(i) and which passes through a target pixel. The equation 
for this line is: 

Ki(Kp)=( Kp- KoV <t> + Ko, Ko <= K p <- Ki. 

Where Kp is the column index of the target pixel, Ko is the pivot pixel and Kj is the end 
of the transition segment. Sub-pixel interpolation is needed when Ki(Kp) is not an 
integer. The phase (J> has a large influence on the value of Ki(Kp). For example, referring 
to Table I, the equation of the line that described the feature frontier is Y(K) - (Kp-<5)/l 1. 

25 Sub-pixel interpolation arises in two situations: 

i) when Y(l) is not an integer; and 

it) when (Kp, <J>) is too close to the target pixel boundary. 

30 Condition ii) is most dramatic when, for example, <b = 0.5, and the number of pixels 
between Ko and Ki is a small even number. In what follows we will focus on i), 
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Let <1> be the phase between 0 and 1 . Let K p be the column position of the target pixel. Let 
us assume that matching segments have been shifted so the transitions regions are aligned 
to within one pixel. The target pixel can be thought of as lying anywhere within the four 
5 bounding pixels. The weights of the four bounding pixels must be chosen so that they 
coincide with the location of the target pixel. 

Given the pixels values P(l,l), P(l,2), P(2,l) and P(2,2), the phase d>, we want to 
compute the target by. interpolation between P(l,l) and P(l,2) and between P(2,l) and 
10 P(2,2). Therefore, we must compute: 

Z,= (l-a)*P(l,l)-*-a*P(1.2) 
Z 2 =(l-b)*P(2 > l)+b*P(2 J 2) 
O Target = (1 -d>)*Zi+**Z2 

;! Where a and b are weights such that 0 <= a, b <= 1. These conditions will give a target 

ft! pixel with twice the desired intensity. In addition, sub-pixel interpolation, as written 

03 

~y above, is a two-step procedure. We can rewrite the above so that 

P 

03 20 Target - (l-<J))*((l-a)*P(l,l)+a*P(l,2))+ <|)*((Ub)*P(2,l>4-b*P(2,2)). 

M 

CI This is interpolation that takes one step. It allows the target pixel to reside anywhere 

^ within the four corner points (P(l,l), P(1,2), P(2,l), P(2,2)), In general, the weights a and 
b are related to the phase <J> which can lead to simplifications, but there are still problems 

25 with which we must contend. Namely, 

i) additional run-time multiplies are required; 

ii) the resultant weights may no longer have unity gain. 

30 Two alternative solutions are suggested. 
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A first alternative may use an existing low pass filter (on chip) to approximate the 
location of the target pixel. Then, by independently flipping the filter weights associated 
with pixels P(1,0) and P(U), P(2,l) and P(2,2), it is possible to reach a much larger 
region of potential target pixel locations. The phase <J> will determine the tap weights, and 
5 Ki(Kp) determines the horizontal intercept, which in turn, determines whether the weights 
should be flipped. Flipping the weights requires multiplexors. 

A second alternative is similar to the first alternative, but it employs a dedicated m by n 
coefficient matrix whose entries act to quantize the square [0,1] X [0,1] into discrete 
10 cells. The fractional portion of the shift, i.e. Kl(Kp) - |_K.l(Kp)J, where UJ is the largest 
integer less than t, and the phase <J> are used to address the coefficient cell in which the 
target resides. Table 3 shows a small table of coefficients that can be used for sub-pixel 
interpolation. 

O 
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Index 


w 0 


w i 


w 2 


w 3 


0 


4/8 


0 


4/8 


0 


1 


4/8 


0 


1/8 


3/8 


2 


4/8 


0 


2/8 


2/8 


3 


4/8 


0 


3/8 


1/8 


4 


4/8 


0 


0 


4/8 



Ul IS Tabic 3: Sub-pixel interpolation weights with pivot pixel located at w0 for i = 0-5- 

The weights w 0> w h w 2 and w 3 in Table 3 coincide with pixels P(l,l), P(l»2), P(2,l) and 
P(2,2) in Figure 1. The index entry in Table 3 and Figure 13 shows how to attain various 
targets. Due to a horizontal shift the pixels P(2,l) and P(2,2) may actually correspond to 
20 P(2,l+R) and P(2,2+R) for R>0. 

The second alternative is preferred because it provides more control over the exact 
location of the target. 
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In conjunction with Table 3, we can see how the weights and pivot pixel P(0 f 0) are used 
in sub-pixel interpolation. The pivot pixel provides the total contribution for row 1. The 
phase of the target pixel is 180 degrees because w 0 +w 1=^2+^3. Table 3 can be further 
reduced to three rows and half the number of columns by exploiting the property of skew 
5 symmetry. Added multiplexors will be required as a consequence to independently toggle 
the weights, 

The effect of phase is an important input into the Alignment Controller 310. In this 
section two examples are given that demonstrate how to compute bounding and target 
10 pixels. We show how the relative shifts are generated with and without the use of sub- 
pixel interpolation. 
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Let Kp be the current column index of the interpolated pixel. Let AS be the accumulated 
shift, S be the shift and TS be the total shift. In order to determine the shift required for a 
15 given Kp, we must solve for Y(Kp)=l. Then, 

K,(K P H Kp- Ko )/<}> + Ko, Ko <= K p <= K t 

Here Kt is the beginning or end of the transition segment at row 2. 

20 Example 1: Segment alignment 

In this example, Ko=6, K,=l 7 so TS=1 7-6=1 1, <H).25; RS=TS=1 1, AS=0. 





S 


RS 


AS 


6 


0 


11 


0 


7 


4 


7 


4 


8 


4 


3 


8 


9 


3 


0 


11 



Table 4 Computation of relative shifts for Table 3 with <|> = 0.5 (1 80 degrees) 



25 The details of the computation are as follows: 
Kp = 6. 
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Ki(6) - (6-6)/0.25+6"6; 
S=Ki(6)- Ko -AS=0. 
RS-RS-S-11; 
AS=AS+S=0; 

5 

K„ = 7. 

Ki(7) = (7-6)/0.25+6=10; 
S=Ki(7)- Ko-AS=4. 
RS=RS-S=7; 
10 AS=AS+S=0+4=4; 

Kp = 8. 

K l (8) = (8-6)/0.25+6=14; 
S=Ki(8)- Ko-AS=4. 
15 RS=RS-S=7-4=3; 
AS=AS+S=4+4=8; 

Ki(9) « (9-6)/0.25-H5=18> Ki=17 so S= Ki-14=17-14=3. 
20 S=3 

RS=RS-S = 0; 
AS=AS+S=4-^4=U; 

In this example, Transition Segment alignment requires two shifts of four pixels and one 
25 shift of three pixels. The third shift brings the intercept location (for row 2) past the 
segment boundary, and so the final shift is three, and not four. After four successive shifts 
the matched segments are aligned. These steps are summarized in Table 4 above. 

The Alignment Controller entries are shown in Table 5 for Alignment Controller entries 
30 for phase adjusted feature alignment for Table 3 for <j> - 0.25 (90 degrees). 
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Shift Row 1 


0 


0 


0 


0 


Shift Row 2 


0 


4 


4 


3 



w 

.ft 

fU 
CO 

fU 



Ul 
0 
H 



10 



20 



Example 2: Segment alignment with sub-pixel interpolation 

In this example we introduce the notion of sub-pixel interpolation. This is required when 
the relative shifts required for alignment are not whole numbers. 

Let Ko=6 and K|=17, then TS= 17-6=11, RS=11, AS=0 and <M>-22. Without loss of 
generality, we assume that Ko is the origin. 



Kp-0. 
ASo=0; 
rAS 0 l=0; 
So=TaSo1=0 



15 RSo=ll 



Kp=l. 

ASi«AS 0 + 1/0.22 = 4.5454 
Xi«?ound(AS|) = 5 
Pi=Xi-Xo=5-0=5 
Si^mfPhRSo^inCS,! 1)=5 
RS|=RS 0 -S,=1 1-5=7; 



Kp-2. 

25 AS 2 « AS t + 1/0.22 = 9.0909 
X2=round(AS2) = 9 

P 2 =X r X,=9-5=4 
S2=min(P 2 ,RSi)«min(4,7)=4 
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RS 2 ~RS,-S2=»7-4=3; 
K p = 3. 

AS 3 = AS 2 + 1/0.22= 13.6363 
5 X 3 =round(AS 2 ) = 14 
P 3 ~X 3 -X 2 = 14-9=5 
S3 aa min(5,RS)=min(5 ) 3)=3 
RS 3 =RS 2 -S 3 =3-4=0; 
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At Kp = 1, a shift of 4.5454 is needed, but it is not possible to shift by this amount In 
order to produce an effective shift of 4.5454, we must first shift by 4, and use sub-pixel 
interpolation to produce an effective shift of 0.5454. The target point is given by: 

Target = (U<t>)*P(l,l)+ 4>*((K0.5454)*P(2,1)+0.5454*P(2,2)). 
As <M).22, we have 
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Target = 0.78P(l,l>+O.lP(2,l)-K).12P(2 5 2). 

20 As discussed with respect to Table 4, the cell into which (Y(l), <t>) falls is used to address 
the weights. Depending on the latency with which the weights are chosen, it may be 
necessary to store the address of the weights used for sub-pixel interpolation along with 
the relative shift. 



25 Table 6 lists the relative incremental shifts needed to achieve transition segment 
alignment for the first matched segment. 



Index (Kp) 


5 


6 


7 


8 


9 


Row 1 
(Shift) 


0 


0 


0 


0 


0 
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Row 2 


0 


4 


5 


4 


1 


(Shift) 













Table 6: Shift Table for Alignment Controller entries for phase adjusted feature alignment for $ « 

0.22 (79.2 degrees). 

In this section we will examine the behavior of one aspect of the shift computation in 
5 more detail. The relative shift is governed by the equation 

Ki(K p )=( Kp- Ko )/ 4 + Ko, Ko <= K p <= Ki. 

We may, without loss of generality, assume that Ko=0. The equation can be simplified to 
10 read 

Ki(KpHV4>, 0<= K P <~K X . 

Clearly, in order to compute the required (relative) shift, the phase § must be inverted. 
15 Inversion is expensive, so instead, inverted values of the phase arc quantized and stored 
as shown in Table 7 where we have cut the interval 0 to 1 into 8 segments of equal width. 
Each row contains an approximation to the inverted phase for a segment. The numbers 
are stored on chip in binary format. 



Quantized Phase <J» 


Inverse of Quantized Phase 1/<J> 


(0,1/8] 


8 


(1/8,2/8] 


4 


(2/8,3/8] 


2.66666 


(3/8,4/8] 


2 


(4/8,5/8] 


1.6 


(5/8,6/8] 


1.33333 


(6/8,7/8] 


1.14285 


(7/8,1) 


0 



20 Table 7: Inverted phase shift table. 
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The inverted phase table is stored in fixed-point representation, but can be also stored in 
another representation, such as floating point. The inverted phase is the required relative 
shift used to align the features. The Accumulated Shift (AS) is used to sum up successive 
shifts and is also used to determine when shifting should cease. Shifting ceases when the 
subsequent pivot pixel is encountered. Logic is used to determining the relative shift at 
the boundary conditions, namely, at 0 and at 1. For example, if the phase is 0, then we 
may to forgo a shift entirely, or alternatively, we could decide to shift by Kr Ko in one 
step. When the phase is 1, similar logic can decide the desired amount of the relative 
shift. 

Alignment Recovery (AR) is the process of bringing the filter back to vertical. Restoring 
the default orientation means that the relative shifts initially used to align transitions 
segments must now be undone. This occurs immediately following the alignment of a 
Q transition segment The alignment recover process is opposite to the alignment process. 

15 The subsequent pivot pixel is used to steer the filter towards a nominal orientation. 
W Thereafter, barring the emergence of other matched transition segments, the default step 

jy size (time increment) is one pixel horizontally (vertically) when interpolating vertically 
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10 



(horizontally). 

20 Other anticipated alignment strategies can be envisioned that generalize the role of the 
pivot pixel. For instance, rather than to use the pivot pixel repeatedly in the transition 

HI 

p region, we can also stagger the pivot about a number of pixels both before and after the 

pivot pixel. This is sometimes useful when smoother transitions are required between 
features. 

25 

An important element of the foregoing discussion is the role of noise. There are many 
types of noise with which we must contend during the feature extraction, comparison and 
alignment processes. For instance, we can refer to pulse noise which is noise of a certain 
magnitude and duration. Pulse noise is not a feature and as such should not influence the 
30 behavior of the Alignment Controller. 
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A Feature Extractor will search for Pulse noise. This kind of noise rejection serves to 
establish the degree to which the current image is noisy. This information can be 
incorporated into the selection of filters and threshold dynamically rendering the entire 
chip truly adaptive in nature. 

It is important to identify noise either using a separate Feature Extractor or as part of the 
current feature extractors so that noise is not classified as a feature. This may result in a 
Matched Table entry that may result in a possible unwanted shift alignment. 



10 Texture Noise is characterized by frequent changes in the direction (changes in A) of the 
image data on a row of a specific size and duration. These changes may be considered 
visually insignificant because they are of short duration or of small magnitude as 
measured by user-defined thresholds. A Feature Extractor can be designed and used to 

□ determine the degree to which the surface is textured. 

% 15 

W Referring to Figure 15 there is shown a circuit diagram depicting the relationship 

ry between the major functional components described above. The Feature Extractor (FE) is 

comprised of N storage locations for the Segment Table each containing: a starting pixel 
intensity; a starting pixel index; an ending pixel intensity; and an ending pixel location. 
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Each specific feature of interest, which is to be identified, requires a specific state 
machine. The Feature Comparator (FC) consists of M storage locations for the Matched 
Table containing: matched trend segments in pairs and a flag indicating a non-transition 
segment The feature comparator implements the match acquisition process flow 
25 described with reference to Figure 11. The Alignment Controller (AC) consists of P 
storage locations for the Alignment Table each containing: the relative shift needed for 
row 1 or row 2; the relative positions at which the relative shift is to occur, information to 
choose the correct weights for sub-pixel interpolation; A phase inversion table for relative 
shift computation and accumulators and decision circuitry for alignment decisions. 

30 
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The circuit to implement the relative shifts consists of fixed-point division, addition and 
logic circuits for implementing the flow described with reference to Figure 14. 



5 The terms and expressions which have been employed in the specification are used as 
terms of description and not of limitations, there is no intention in the use of such terms 
and expressions to exclude any equivalents of the features shown and described or 
portions thereof, but it is recognized that various modifications are possible within the 
scope of the claims to the invention. 



5 
ni 

m 

m 
03 

M 
Ul 



26 



m-i eso/eeo d zao-i 



dll ujinowiQ neeimjBH ue)|se^oild WLl l00Z-Sl-3flV 



